Uncertainty in quantitative health impact modelling
MRC Biostatistics Unit, University of Cambridge
Uncertainty in HIA models: Summary of lecture
Parametric models — definitions
Quantifying parameter uncertainty using probability distributions
Questions we can answer better by quantifying uncertainty
Uncertainty analysis — how confident are our results, how strong is the evidence?
Sensitivity analysis — how would results change if part of the model changed?
Value of Information (VoI) analysis — what parts of the model do we need better evidence for?
Quantitative models
Models approximate process that generates some output of interest, helping to inform decision-making.
Example: ITHIM health impact model [ TODO borrow picture ]
inputs: travel data, air pollution, physical activity, injuries, + scenarios of change
outputs: population health (and changes under scenarios)
These generally involve parameters. What are these?
Parametric models
Model input parameters: examples
background exposure [measures of average e.g. air pollution, physical activity] in an area
relative risk for [health outcome] given [change in AP / exposure]
Parameters are representations of knowledge about a (real or imagined) population.
- Usually conceive as summaries of individual quantities over a large/infinite “population”.
Model input parameters \(\rightarrow\) model output quantities.
Model outputs are usually summaries over populations
Parameter uncertainty
Knowledge is often uncertain \(\rightarrow\)
parameters are uncertain
conclusions from models are uncertain
Individual variability versus uncertainty about knowledge
Example parameters, for some population
Background exposure to PM2.5.
Risk of death within one year.
We may be sure about the values of these. Even so…
Exact exposures will be different for each individual
Some individuals will die within a year, some won’t
But we may be uncertain about the parameter values.
Individual variability can never be removed, but parameter uncertainty can be reduced with better knowledge
Parameters are generally uncertain
Two broad reasons for parameter uncertainty
summary of a limited population
population is different from the one we want
- Discuss examples of this Models are also uncertain (“structural” uncertainty, “all models are wrong, but some are useful”…)
- Ideally quantify as many uncertainties as possible inside the model
Why does uncertainty matter?
[ interact ? ]
We’ve built best model we can. Just report our “best estimate” to the decision maker?
Decision makers need good evidence to change practice — models should indicate strength of evidence for result
What about the future - we may be able to get better evidence - what research should be done?
Here we will cover quantitative methods for assessing uncertainty. In particular, probabilistic methods.
- though there is a broader field of appraising evidence, engaging with stakeholders or experts etc…
Questions relevant to uncertainty quantification
Uncertainty analysis: about strength of evidence
What range of outputs are plausible, given current evidence?
Which parameters most influence the output uncertainty?
Sensitivity analysis: “what if…”
- What if parameter took the value \(b\) instead of \(a\) — how would the output change?
Value of Information analysis
- How much would the model improve if we got better information?
Which parameter should we get better information on?
How to quantify uncertainty
Two broad approaches
Statistical analyses of data (your own, or published analyses) giving point and interval estimates or standard errors
Judgements (informal or from structured elicitation), e.g.
point estimate (best guess) for parameter value
credible interval e.g. “I judge that the parameter is between \(a\) and \(b\), with 95% confidence”
Our goal in each case is to obtain probability distributions for parameters
- rigorous basis for uncertainty and sensitivity analysis
Probability distributions
A full probability distribution
For any pair of values \((a,b)\), we can deduce the probability that the parameter is between \(a\) and \(b\).
Statistical analyses: Bayesian methods
In statistical models, data assumed to come from models with parameters…
e.g. number of disease cases is Binomial (population n, underlying prevalence p)
Bayesian methods based around quantifying parameter uncertainties as probability distributions
Prior distribution combined with study data \(\rightarrow\) posterior distribution.
Prior distribution dominates if data are weak
Prior doesn’t matter if enough data
Statistical analyses: frequentist methods
Construct an estimator of parameters based on a finite dataset
Uncertainty quantified by imagining different datasets drawn from the same population
Standard error: variability in estimates between datasets.
Confidence interval: contains true value in 95% of draws.
If dataset is large, can interpret a frequentist analysis as Bayesian: parameter has a normal distribution, defined by estimate and standard error
If dataset is small, Bayesian analyses have the benefit of allowing background information to be included as a prior
Examples of statistical models informing QHIA
Global
disbayes informing incidence, CF
Meta analysis of relative risks
Local
(transport) travel time speeds convert to distances
Road traffic injury models - Poisson
Examples of judgements
Example: MMET values for transport-related cycling
Which of these are relevant to our particular health impact model — pick 6.8 (4, 10) given judgement?
Examples of judgements
Example: PM2.5 concentration. [SHOW APnA Table 3.] We have average and SD for 30 cities in India.
We have average for our city, but we want a measure of uncertainty
Could choose our SD as average of these SDs (?? geometric average (exp(mean(log))) or median?)
But what do we know about how our city compares to these?
Is it worth woing into effort to quantify — start with conservative assessment of uncertainty
Considerations: * size and quality of data underlying * how was it measured * relevance of population * changes over time * changes over space
Where might these come from with no extra info?
Meta-analysis
Formally averaging information from different studies.
Most developed in randomised clinical trials — highly controlled, regulated studies.
Give more weight to more confident studies and/or those closer to our context.
TODO any resources for MA of clinical trials and epidemiology. R package at least
PICTURE of meta-analysis
Uncertainty about a predicted new study – more appropriate than the average
Distributions from credible intervals
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
What is wrong with a distribution like this
Distributions from credible intervals
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
A triangular distribution is a bit more plausible
Distributions from credible intervals
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
A normal distribution is even better
Normal distribution
Used for quantities with unrestricted ranges
Defined by mean \(\mu\) and standard deviation \(\sigma\) (or variance \(\sigma^2\))
95% credible interval is \(\pm 2\) SDs: i.e. width is 4 SDs.: SD easily derived from a CI.
Log-normal distribution
Used for positive-valued quantities.
Normal distribution for the log of the quantity
Example: MMET/h estimate 2.5 (CI 1 to 4).
Transform to log(MMET).
Estimate log(2.5), CI (log(1) to log(4)).
Assign normal with SD = CI width / 4
If the SD is published, but not the CI, e.g. MMET/h estimate \(m=2.5\) (SD \(s=1.4\))?
- Method of moments gets us mean \(\mu\) and SD \(\sigma\) on log scale \(\mu = \log(m/\sqrt{s^2/m^2 + 1}), \sigma= \sqrt{\log(s^2/m^2 + 1)}\)
Beta distribution
Used for quantities between 0 and 1: probabilities, proportions
Two “shape” parameters \(a,b\). Mean: \(a/(a+b)\).
Given an estimate \(m\) and credible interval, could approximate SD \(s =\) CI width/4 and use method of moments: \(a = (m(1-m)/s^2 - 1)m, b = (m(1-m)/s^2 - 1)(1 - m)\)
Example (a): estimate 0.4, 95% CI 0.2 to 0.6
Example (b): estimate 0.04, 95 %CI 0.01 to 0.4
Distribution (b) is a worse approximation to our desired CI width
“width/4” heuristic based on symmetric, normal-shaped distributions.
Could informally calibrate to match desired belief more closely.
See SHELF R package for more sophisticated techniques for fitting distributions
Practical - quantifying uncertainty around parameters
[ reference to ITHIM ]
Doing uncertainty analysis: Monte Carlo simulation
For each \(i = 1, 2, \ldots N\) (enough to give precise summaries)
Simulate parameters \(X_i\) from their uncertainty distributions
Compute the model output \(Y_i = g(X_i)\)
producing a sample from the model outputs \(Y_1, \ldots, Y_N\) [ animate? ]
Summarise the sample to give e.g.
a credible interval for the outputs
probability that e.g. number of deaths \(>\) [important value]
Technical point - uncertainty affects “best estimate” in nonlinear models
Given a model with inputs \(X\) and outputs \(Y = g(X)\), where \(g()\) is some mathematical function.
Inputs \(X\) are uncertain, with expected values \(E(X)\).
What is the expected value of \(Y\)? Can we just plug in the best estimates of the inputs \(g(E(X)\)?
\(E(Y)\) is only equal to \(g(E(x))\) if the function \(g()\) is linear:
\[g(E(x)) = g((X_1 + \ldots + X_n)/n) = \] (if \(g()\) linear…)
\[(g(X_1) + \ldots + g(X_n)) / n = E(g(X)) = E(Y)\]
Most realistic models are **non-linear*: need Monte Carlo simulation to get the true expectation of the output.
One-way sensitivity analysis
Answers questions like “what if [parameter] was actually \(b\) instead of \(a\)”
For each parameter, compare model output with
parameter at a “low value”
parameter at a “high value”
What if all the other parameters are uncertain? Probabilistic one-way sensitivity analysis:
Fix [parameter] at high or low value
Run the model under Monte Carlo analysis, using the uncertainty distributions of the other parameters
Tornado plot TODO nicer
Practical - uncertainty analysis in a model
[ reference to ITHIM ]
Value of Information
Recall
sensitivity analysis: “what if model was a bit different”
uncertainty analysis: “what is strength of evidence in model”
A different (related) question: “what would be the benefit of getting better information”.
This is Value of Information analysis
[Example - connect to a result from probabilistic 1 way SA. Looks like varying ? within plausible range has more of an effect than… so we might want to get better info ]
Value of Information
Given current information \(x\) model output is uncertain. But how much more precise would it get…
if we were to learn some parameter exactly?
- Expected value of partial perfect information
if we conducted a study (say, a survey of 100 people to estimate it
- Expected value of sample information
Helps us to:
set research priorities to reduce uncertainty
design studies (more advanced, not covered here)
Value of Information in terms of health economics
Value of Information methods developed in health economics
Model a health policy decision and its consequences (e.g. health benefits as QALYs vs costs). Parameter uncertainties as probability distributions
Information has value: \(\rightarrow\) reduces parameter uncertainty \(\rightarrow\) more precise model outputs \(\rightarrow\) better informed policy-making \(\rightarrow\) health benefits
Not previously used much in health impact modelling, where models used more for scenarios than policies
How can we define “value” if we don’t model a health policy?
Precision as value
Define value as “precision of estimate” (e.g. health impact of scenario)
How much will the variance (or SD, or credible interval width) be expected to reduce if we got better information?
More precise estimates (implicitly) assumed to ultimately lead to benefits
If needed, trade off informally with costs of research (willingness to pay for more precise estimates? Not covered here)
How exactly to do these computations?
Computing expected value of partial perfect information (EVPPI)
Model \(Y = g(X_1, X_2,.. )\) with some (scalar) output \(Y\) and multiple inputs \(X_r\)
Do uncertainty analysis: define distributions for each \(X_r\), obtain distribution for \(Y\) via Monte Carlo simulation.
\(var(Y)\): variance of model output under current information
Definition of EVPPI - expected reduction in this variance if we were to learn the exact value of \(X_r\) (say)
\[var(Y) - E_x(var(Y | X_r = x))\]
We don’t know the value of \(X_r\) when we calculate this — so we must take the expectation over possible values \(x\) (using the distribution we defined)
Illustration of EVPPI computation
Take the Monte Carlo sample, with all parameters uncertain
Regression of \(Y\) = model output versus \(X\) = parameter of interest
Expected reduction in variance of \(Y\) if we learnt \(X\)
\[var(Y) - E_x(var(Y | X = x))\]
We don’t know \(X\) so we average over our uncertainty
\(var(Y)\): variance of \(Y\)
\(E_x(var(Y | X = x))\): residual variance: mean of (“observed” - fitted) over different \(x\)
Spline regression for computing EVPPI
Regression function of output on input will not necessarily be linear.
Spline regression (automated choice of smoothness, works well enough in this context)
lm(Y ~ X) # linear model
library(mgcv)
gam(Y ~ X) # spline ("generalized additive") modelThe voi package can take care of extracting ingredients needed for the EVPPI computation (see the practivel)
library(voi)
evppivar(outputs, inputs)EVPPI for more than one parameter at once
Example: dose-response curve governed by four different parameters \(\alpha\), \(\beta\), \(\gamma\), \(\tau\). Estimate the expected value of jointly learning all four of them (hence learning the dose-response curve)
Regression model with four predictors, e.g.
gam(Y ~ alpha + beta + gamma + tau) More advanced regression models available, which automatically choose the best fitting regression function for Y given the predictors (and their interactions): see the practical about the voi package
Presenting EVPPI results
Impact of different parameters on the output uncertainty, presented as
Uncertainty that would remain in the output \(Y\) if we knew \(X\)
as a variance, \(var(Y) - EVPPI_X\)
…or standard deviation (square root of this)
…or rough credible interval (\(\pm 2\) remaining standard deviation)
or proportion of uncertainty (variance) in \(Y\) explained by \(X\)
Value of Information: further topics
We will never get perfect information, but we may get better information via a new research study.
Expected value of sample information is the expected value of a study of a particular design / sample size
Slightly harder to define and compute
Previously used in health economic decision models, but not (yet!) health impact models
See
voipackage and references there for some examples.
Value of Information: further topics
Here we have just talked about “value” as variance of outputs
Value of Information also used widely for health economic decision models
Model chooses policy with the highest net benefit (or lowest loss)
“Value” is the net benefit (NB) itself
Value of information is NB(better information) - NB(current information)
See voi package and references there for some examples.
Practical session
… Use of “voi” package
Group discussion of advanced topics
What uncertainty/sensitivity questions do you want to answer in your own work?
Do you have the tools to do this?